BUG: make `read_csv` be able to read large floating numbers into float #62542

Alvaro-Kothe · 2025-10-02T02:32:28Z

Xref BUG: read_csv() fails to detect floats larger than 2.0^64 #51295 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Notes

This doesn't fully close the issue because it only addresses big floating numbers. Large integers are still read as strings (This is tested in pandas/tests/io/parser/common/test_ints.py).
Large floats may experience precision loss due to floating-point representation.

Implementation details

Skips the integer parsing if it detects a potential floating number by checking for the presence of:

decimal separator
exponent sign (e/E)

pandas/_libs/parsers.pyx

mroeschke · 2025-10-03T17:39:17Z

pandas/_libs/parsers.pyx

+            if na_filter and kh_get_str_starts_item(na_hashset, word):
+                continue
+
+            if self.parser.decimal in word or b"e" in word or b"E" in word:


Do we know if a word like "NaN" would reach this point?

Is it naive to try float(word) to check if word is a float?

Is it naive to try float(word) to check if word is a float?

I don't think it will work because integers can be cast into floats.

Do we know if a word like "NaN" would reach this point?

I added a print statement before the return True and ran the test suite in pandas/tests/io/parser. The word "NaN" didn't reach. The words that flagged the column as float are shown below.

It is unintended that strings that contain the letter "e" flags the column as "float", but since this function only serves to skip int parsing, the result isn't that problematic.

Marking as float because of b'0.0' Marking as float because of b' 0.0000' Marking as float because of b' 0.0100' Marking as float because of b'0.056674973' Marking as float because of b' 0,1 ' Marking as float because of b'0,1' Marking as float because of b'0.1' Marking as float because of b'01:10:18.300' Marking as float because of b'0.18905338179353307' Marking as float because of b'0.2' Marking as float because of b'0.212036' Marking as float because of b'0.2140' Marking as float because of b'0.2616121342493164' Marking as float because of b'0.31814660061537436' Marking as float because of b'0355626618.16711' Marking as float because of b'-0.364216805298' Marking as float because of b'-0.41306354339189344' Marking as float because of b'0.43263079080478717' Marking as float because of b'0.5' Marking as float because of b'0.5165781941249967' Marking as float because of b'-0.5227484414807474' Marking as float because of b'-0.689265' Marking as float because of b'-0.692787' Marking as float because of b' 0.8100' Marking as float because of b'0.980268513777' Marking as float because of b' ,1 ' Marking as float because of b' -,1 ' Marking as float because of b' _1, ' Marking as float because of b' _1,_ ' Marking as float because of b' 1, ' Marking as float because of b' 1_, ' Marking as float because of b',1' Marking as float because of b'-,1' Marking as float because of b'_1,' Marking as float because of b'_1,_' Marking as float because of b'1,' Marking as float because of b'1.' Marking as float because of b'1_,' Marking as float because of b' -1,0 ' Marking as float because of b'-1,0' Marking as float because of b'1.0' Marking as float because of b'10,' Marking as float because of b'10.' Marking as float because of b' 1_000,000_000 ' Marking as float because of b'1_000,000_000' Marking as float because of b'1.00361' Marking as float because of b'1032.43' Marking as float because of b'1.050000000000000044408921' Marking as float because of b'10E-100000' Marking as float because of b'10E-617' Marking as float because of b'10E-99999999999999999' Marking as float because of b'10E-999999999999999999' Marking as float because of b'10E999999999999999999' Marking as float because of b'1.1' Marking as float because of b'1.100000000000000088817842' Marking as float because of b'1.12551' Marking as float because of b'1.149999999999999911182158' Marking as float because of b'-1.15973806169' Marking as float because of b'1.199999999999999955591079' Marking as float because of b' ,1__2 ' Marking as float because of b' --1,2 ' Marking as float because of b' 1,_2 ' Marking as float because of b',1__2' Marking as float because of b'--1,2' Marking as float because of b'1,_2' Marking as float because of b'1.2' Marking as float because of b'1.200' Marking as float because of b' 1,2_1 ' Marking as float because of b'1,2_1' Marking as float because of b' 1,2,2 ' Marking as float because of b'1,2,2' Marking as float because of b' 1_234,56 ' Marking as float because of b'1_234,56' Marking as float because of b'12345,67' Marking as float because of b' 1_234,56e0 ' Marking as float because of b'1_234,56e0' Marking as float because of b'1234E+0' Marking as float because of b'1.25' Marking as float because of b' -1,2e0 ' Marking as float because of b'-1,2e0' Marking as float because of b' 1,2e_1 ' Marking as float because of b'1,2e_1' Marking as float because of b' 1,2E-1 ' Marking as float because of b' 1,2E1 ' Marking as float because of b'1,2E-1' Marking as float because of b'1,2E1' Marking as float because of b' 1,2e1_0 ' Marking as float because of b'1,2e1_0' Marking as float because of b' 1,2e-10e1 ' Marking as float because of b'1,2e-10e1' Marking as float because of b'1.300000000000000044408921' Marking as float because of b'1.350000000000000088817842' Marking as float because of b'1352171357E+5' Marking as float because of b'1.399999999999999911182158' Marking as float because of b'1.4' Marking as float because of b'1.449999999999999955591079' Marking as float because of b'14.7674' Marking as float because of b'1.5' Marking as float because of b'1521,1541' Marking as float because of b'1.550000000000000044408921' Marking as float because of b'1.600000000000000088817842' Marking as float because of b'1.649999999999999911182158' Marking as float because of b'1.700000000000000177635684' Marking as float because of b'1.75' Marking as float because of b'179.71425' Marking as float because of b'1.800000000000000044408921' Marking as float because of b' 18446744073709551616.0' Marking as float because of b' 18446744073709551616.5' Marking as float because of b'1.850000000000000088817842' Marking as float because of b'187101,9543' Marking as float because of b'1.899999999999999911182158' Marking as float because of b'1917.09447' Marking as float because of b'1.950000000000000177635684' Marking as float because of b' 1a_2,1 ' Marking as float because of b'1a_2,1' Marking as float because of b' ,1e ' Marking as float because of b' -,1e ' Marking as float because of b',1e' Marking as float because of b'-,1e' Marking as float because of b'1E' Marking as float because of b' +1,e0 ' Marking as float because of b' -1,e0 ' Marking as float because of b'+1,e0' Marking as float because of b'-1,e0' Marking as float because of b' +1e+0 ' Marking as float because of b' +1e0 ' Marking as float because of b' -_1e0 ' Marking as float because of b' -1e0 ' Marking as float because of b' _1e0 ' Marking as float because of b'+1e+0' Marking as float because of b'+1e0' Marking as float because of b'-_1e0' Marking as float because of b'-1e0' Marking as float because of b'_1e0' Marking as float because of b' +,1e1 ' Marking as float because of b' +1e-1 ' Marking as float because of b' -,1e1 ' Marking as float because of b'+,1e1' Marking as float because of b'+1e-1' Marking as float because of b'-,1e1' Marking as float because of b' 1e11,2 ' Marking as float because of b'1e11,2' Marking as float because of b' 1,e1_2 ' Marking as float because of b'1,e1_2' Marking as float because of b'2.' Marking as float because of b'2.0' Marking as float because of b'2.2' Marking as float because of b' 2.2100' Marking as float because of b'225.874' Marking as float because of b'2,334.01' Marking as float because of b'2.334,01' Marking as float because of b'240.000' Marking as float because of b'243.164' Marking as float because of b'2456026.548822908' Marking as float because of b'2.5' Marking as float because of b'252.373' Marking as float because of b' 260.0000' Marking as float because of b' 280.0000' Marking as float because of b' 2.8100' Marking as float because of b'2e' Marking as float because of b'3.' Marking as float because of b'314.11625' Marking as float because of b' 32.0' Marking as float because of b' 32e0' Marking as float because of b' 3.2e1' Marking as float because of b' 3.2e-80' Marking as float because of b' 3.2e80' Marking as float because of b'3.3000000000000003' Marking as float because of b'330.65659' Marking as float because of b'3.4' Marking as float because of b'344.98370' Marking as float because of b'3.5' Marking as float because of b'3.68573087906' Marking as float because of b' 36893488147419103232.3' Marking as float because of b'3E' Marking as float because of b'4.' Marking as float because of b'412.166' Marking as float because of b'41.605' Marking as float because of b'42e' Marking as float because of b'4.5' Marking as float because of b'45.' Marking as float because of b'45e-1' Marking as float because of b'4,738797819' Marking as float because of b'4.8' Marking as float because of b'5.' Marking as float because of b'5.1' Marking as float because of b'632E' Marking as float because of b'64.0' Marking as float because of b'65248E10' Marking as float because of b' .67' Marking as float because of b'70.06056' Marking as float because of b' 7.2000' Marking as float because of b'73.48821' Marking as float because of b'7.5' Marking as float because of b' .78' Marking as float because of b'80.000' Marking as float because of b' .81' Marking as float because of b'.86' Marking as float because of b' .88' Marking as float because of b'-9,1' Marking as float because of b'apple' Marking as float because of b'DEF' Marking as float because of b'e' Marking as float because of b' e11,2 ' Marking as float because of b'e11,2' Marking as float because of b'e,d' Marking as float because of b'EEE' Marking as float because of b'e\n d' Marking as float because of b'example\n sentence\n two' Marking as float because of b'False' Marking as float because of b' hello' Marking as float because of b'hello' Marking as float because of b'hello\nthere' Marking as float because of b'"hello world"' Marking as float because of b'http://www.ikea.com/se/sv/catalog/categories/departments/living_room/10475/?se%7cps%7cnonbranded%7cvardagsrum%7cgoogle%7ctv_bord' Marking as float because of b'Hugo Chavez' Marking as float because of b'Hugo Ch\xc3\xa1vez' Marking as float because of b'Hugo Ch\xc3\xa1vez Fr\xc3\xadas' Marking as float because of b'Hugo Rafael Chavez Frias' Marking as float because of b'index' Marking as float because of b'index1' Marking as float because of b'Iris-setosa' Marking as float because of b'King of New York (1990)' Marking as float because of b"line '21' line 22" Marking as float because of b"line '21\n' line 22" Marking as float because of b'line 21\nline 22' Marking as float because of b"line '21\n' \r\tline 22" Marking as float because of b"line \n'21' line 22" Marking as float because of b'None' Marking as float because of b'one' Marking as float because of b'President' Marking as float because of b'qwer' Marking as float because of b'Raphael' Marking as float because of b'rectangular' Marking as float because of b'red' Marking as float because of b'rez' Marking as float because of b'SELL' Marking as float because of b'Sixth Man, The (1997)' Marking as float because of b'SLAGBORD, "Bergslagen", IKEA:s 1700-tals series' Marking as float because of b'somedatasomedatasomedata1' Marking as float because of b'tables' Marking as float because of b' test' Marking as float because of b'test' Marking as float because of b'test \x1a test' Marking as float because of b'True' Marking as float because of b'TRUE' Marking as float because of b'Venezuela' Marking as float because of b'\xe3\x81\x9d\xe3\x81\xae\xe7\xb6\x9a\xe7\xb7\xa8\xe3\x81\xa7\xe3\x81\x82\xe3\x82\x8b\xe3\x80\x8e\xe6\x8c\x87\xe8\xbc\xaa\xe7\x89\xa9\xe8\xaa\x9e\xe3\x80\x8f\xe3\x81\xab\xe3\x81\x8a\xe3\x81\x84\xe3\x81\xa6\xe3\x81\xaf\xe3\x80\x8c\xe4\xb8\x80\xe3\x81\xa4\xe3\x81\xae\xe6\x8c\x87\xe8\xbc\xaa\xef\xbc\x88the One Ring\xef\xbc\x89\xe3\x80\x8d\xe3\x81\xae\xe4\xbd\x9c\xe3\x82\x8a\xe4\xb8\xbb\xe3\x80\x81\xe3\x80\x8c\xe5\x86\xa5\xe7\x8e\x8b\xef\xbc\x88Dark Lord\xef\xbc\x89\xe3\x80\x8d\xe3\x80\x81\xe3\x80\x8c\xe3\x81\x8b\xe3\x81\xae\xe8\x80\x85\xef\xbc\x88the One\xef\xbc\x89[1]\xe3\x80\x8d\xe3\x81\xa8\xe3\x81\x97\xe3\x81\xa6\xe7\x99\xbb\xe5\xa0\xb4\xe3\x81\x99\xe3\x82\x8b\xe3\x80\x82\xe5\x89\x8d\xe5\x8f\xb2\xe3\x81\xab\xe3\x81\x82\xe3\x81\x9f\xe3\x82\x8b\xe3\x80\x8e\xe3\x82\xb7\xe3\x83\xab\xe3\x83\x9e\xe3\x83\xaa\xe3\x83\xab\xe3\x81\xae\xe7\x89\xa9\xe8\xaa\x9e\xe3\x80\x8f\xe3\x81\xa7\xe3\x81\xaf\xe3\x80\x81\xe5\x88\x9d\xe4\xbb\xa3\xe3\x81\xae\xe5\x86\xa5\xe7\x8e\x8b\xe3\x83\xa2\xe3\x83\xab\xe3\x82\xb4\xe3\x82\xb9\xe3\x81\xae\xe6\x9c\x80\xe3\x82\x82\xe5\x8a\x9b\xe3\x81\x82\xe3\x82\x8b\xe5\x81\xb4\xe8\xbf\x91\xe3\x81\xa7\xe3\x81\x82\xe3\x81\xa3\xe3\x81\x9f\xe3\x80\x82' Marking as float because of b'YES'

I will update the verification to make it return early if it finds a word that isn't numeric.

Co-authored-by: Matthew Roeschke <[email protected]>

Alvaro-Kothe · 2025-10-03T20:33:01Z

Here are the benchmarks on io.csv comparing the latest changes (without 380e6ec) against main. The worst performance hit was an increase of 1.43ms to 4.04ms on io.csv.ReadUint64Integers.time_read_uint64

| Change   | Before [c8213d16] <main>   | After [7794b73c] <fix/read_csv-large-num>   | Ratio   | Benchmark (Parameter)                                                                   |
|----------|----------------------------|---------------------------------------------|---------|-----------------------------------------------------------------------------------------|
|          | 17.3±0.3ms                 | 16.4±0.4ms                                  | 0.95    | io.csv.ParseDateComparison.time_read_csv_dayfirst(False)                                |
|          | 3.41±0.02ms                | 3.48±0.09ms                                 | 1.02    | io.csv.ParseDateComparison.time_read_csv_dayfirst(True)                                 |
|          | 18.4±0.1ms                 | 18.2±0.4ms                                  | 0.99    | io.csv.ParseDateComparison.time_to_datetime_dayfirst(False)                             |
|          | 3.56±0.05ms                | 3.58±0.06ms                                 | 1.01    | io.csv.ParseDateComparison.time_to_datetime_dayfirst(True)                              |
|          | 18.2±0.1ms                 | 17.8±0.2ms                                  | 0.98    | io.csv.ParseDateComparison.time_to_datetime_format_DD_MM_YYYY(False)                    |
|          | 3.37±0.04ms                | 3.39±0.02ms                                 | 1.00    | io.csv.ParseDateComparison.time_to_datetime_format_DD_MM_YYYY(True)                     |
|          | failed                     | failed                                      | n/a     | io.csv.ReadCSVCParserLowMemory.peakmem_over_2gb_input                                   |
|          | 901±30μs                   | 896±3μs                                     | 0.99    | io.csv.ReadCSVCachedParseDates.time_read_csv_cached(False, 'c')                         |
|          | 1.51±0.04ms                | 1.49±0.01ms                                 | 0.99    | io.csv.ReadCSVCachedParseDates.time_read_csv_cached(False, 'python')                    |
|          | 881±20μs                   | 916±9μs                                     | 1.04    | io.csv.ReadCSVCachedParseDates.time_read_csv_cached(True, 'c')                          |
|          | 1.48±0.04ms                | 1.51±0.01ms                                 | 1.02    | io.csv.ReadCSVCachedParseDates.time_read_csv_cached(True, 'python')                     |
|          | 25.1±0.5ms                 | 24.3±0.8ms                                  | 0.97    | io.csv.ReadCSVCategorical.time_convert_direct('c')                                      |
|          | 223±4ms                    | 224±2ms                                     | 1.01    | io.csv.ReadCSVCategorical.time_convert_direct('python')                                 |
|          | 61.2±0.3ms                 | 60.1±1ms                                    | 0.98    | io.csv.ReadCSVCategorical.time_convert_post('c')                                        |
|          | 152±1ms                    | 150±4ms                                     | 0.99    | io.csv.ReadCSVCategorical.time_convert_post('python')                                   |
|          | 36.8±0.5ms                 | 36.6±0.7ms                                  | 1.00    | io.csv.ReadCSVComment.time_comment('c')                                                 |
|          | 36.9±0.3ms                 | 35.5±1ms                                    | 0.96    | io.csv.ReadCSVComment.time_comment('python')                                            |
|          | 21.5±0.4ms                 | 21.0±0.4ms                                  | 0.98    | io.csv.ReadCSVConcatDatetime.time_read_csv                                              |
|          | 10.3±0.1ms                 | 10.2±0.2ms                                  | 0.99    | io.csv.ReadCSVConcatDatetimeBadDateValue.time_read_csv('')                              |
|          | 6.78±0.04ms                | 6.79±0.1ms                                  | 1.00    | io.csv.ReadCSVConcatDatetimeBadDateValue.time_read_csv('0')                             |
|          | 11.4±0.08ms                | 11.1±0.2ms                                  | 0.97    | io.csv.ReadCSVConcatDatetimeBadDateValue.time_read_csv('nan')                           |
|          | 3.55±0.01ms                | 3.54±0.02ms                                 | 1.00    | io.csv.ReadCSVDInferDatetimeFormat.time_read_csv('custom')                              |
|          | 1.09±0ms                   | 1.11±0.01ms                                 | 1.02    | io.csv.ReadCSVDInferDatetimeFormat.time_read_csv('iso8601')                             |
|          | 895±20μs                   | 907±2μs                                     | 1.01    | io.csv.ReadCSVDInferDatetimeFormat.time_read_csv('ymd')                                 |
|          | 965±2μs                    | 970±2μs                                     | 1.01    | io.csv.ReadCSVDInferDatetimeFormat.time_read_csv(None)                                  |
|          | 4.30±0.06ms                | 4.33±0.07ms                                 | 1.01    | io.csv.ReadCSVDatePyarrowEngine.time_read_csv_index_col                                 |
|          | 47.3M                      | 47M                                         | 0.99    | io.csv.ReadCSVEngine.peakmem_read_csv('c')                                              |
|          | 63.9M                      | 63.4M                                       | 0.99    | io.csv.ReadCSVEngine.peakmem_read_csv('pyarrow')                                        |
|          | 217M                       | 216M                                        | 1.00    | io.csv.ReadCSVEngine.peakmem_read_csv('python')                                         |
| +        | 9.70±0.3ms                 | 15.9±0.1ms                                  | 1.64    | io.csv.ReadCSVEngine.time_read_bytescsv('c')                                            |
|          | 6.84±0.3ms                 | 7.02±0.3ms                                  | 1.03    | io.csv.ReadCSVEngine.time_read_bytescsv('pyarrow')                                      |
|          | 284±7ms                    | 283±8ms                                     | 1.00    | io.csv.ReadCSVEngine.time_read_bytescsv('python')                                       |
| +        | 9.86±0.4ms                 | 16.0±0.1ms                                  | 1.62    | io.csv.ReadCSVEngine.time_read_stringcsv('c')                                           |
|          | 7.52±0.3ms                 | 7.39±0.2ms                                  | 0.98    | io.csv.ReadCSVEngine.time_read_stringcsv('pyarrow')                                     |
|          | 281±5ms                    | 285±5ms                                     | 1.01    | io.csv.ReadCSVEngine.time_read_stringcsv('python')                                      |
|          | 785±9μs                    | 791±5μs                                     | 1.01    | io.csv.ReadCSVFloatPrecision.time_read_csv(',', '.', 'high')                            |
|          | 1.85±0.03ms                | 1.83±0.02ms                                 | 0.99    | io.csv.ReadCSVFloatPrecision.time_read_csv(',', '.', 'round_trip')                      |
|          | 822±20μs                   | 790±7μs                                     | 0.96    | io.csv.ReadCSVFloatPrecision.time_read_csv(',', '.', None)                              |
|          | 1.10±0.01ms                | 1.10±0.01ms                                 | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv(',', '_', 'high')                            |
|          | 1.10±0ms                   | 1.10±0ms                                    | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv(',', '_', 'round_trip')                      |
|          | 1.10±0.02ms                | 1.10±0ms                                    | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv(',', '_', None)                              |
|          | 788±6μs                    | 784±7μs                                     | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv(';', '.', 'high')                            |
|          | 1.82±0.02ms                | 1.85±0.02ms                                 | 1.02    | io.csv.ReadCSVFloatPrecision.time_read_csv(';', '.', 'round_trip')                      |
|          | 790±6μs                    | 790±10μs                                    | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv(';', '.', None)                              |
|          | 1.09±0.01ms                | 1.10±0.01ms                                 | 1.01    | io.csv.ReadCSVFloatPrecision.time_read_csv(';', '_', 'high')                            |
|          | 1.09±0ms                   | 1.09±0ms                                    | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv(';', '_', 'round_trip')                      |
|          | 1.10±0ms                   | 1.10±0ms                                    | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv(';', '_', None)                              |
|          | 2.53±0.05ms                | 2.64±0.02ms                                 | 1.04    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(',', '.', 'high')              |
|          | 2.53±0.07ms                | 2.62±0.01ms                                 | 1.03    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(',', '.', 'round_trip')        |
|          | 2.58±0.02ms                | 2.61±0.01ms                                 | 1.01    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(',', '.', None)                |
|          | 2.05±0.06ms                | 2.13±0.01ms                                 | 1.04    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(',', '_', 'high')              |
|          | 2.02±0.03ms                | 2.13±0.01ms                                 | 1.05    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(',', '_', 'round_trip')        |
|          | 2.07±0.07ms                | 2.13±0.01ms                                 | 1.03    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(',', '_', None)                |
|          | 2.55±0.09ms                | 2.64±0.04ms                                 | 1.04    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(';', '.', 'high')              |
|          | 2.74±0.09ms                | 2.63±0.01ms                                 | 0.96    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(';', '.', 'round_trip')        |
|          | 2.61±0.1ms                 | 2.61±0.02ms                                 | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(';', '.', None)                |
|          | 2.05±0.08ms                | 2.13±0.01ms                                 | 1.04    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(';', '_', 'high')              |
|          | 2.08±0.06ms                | 2.13±0.01ms                                 | 1.02    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(';', '_', 'round_trip')        |
|          | 2.13±0.03ms                | 2.14±0.01ms                                 | 1.00    | io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(';', '_', None)                |
| +        | 5.26±0.3ms                 | 7.77±0.1ms                                  | 1.48    | io.csv.ReadCSVIndexCol.time_read_csv_index_col                                          |
|          | 70.5±0.9ms                 | 72.0±0.7ms                                  | 1.02    | io.csv.ReadCSVMemMapUTF8.time_read_memmapped_utf8                                       |
|          | 0                          | 0                                           | n/a     | io.csv.ReadCSVMemoryGrowth.mem_parser_chunks('c')                                       |
|          | 0                          | 0                                           | n/a     | io.csv.ReadCSVMemoryGrowth.mem_parser_chunks('python')                                  |
|          | 813±10μs                   | 837±30μs                                    | 1.03    | io.csv.ReadCSVParseDates.time_baseline('c')                                             |
|          | 1.06±0.03ms                | 1.00±0.01ms                                 | 0.94    | io.csv.ReadCSVParseDates.time_baseline('python')                                        |
|          | 3.04±0.01ms                | 3.04±0ms                                    | 1.00    | io.csv.ReadCSVParseSpecialDate.time_read_special_date('hm', 'c')                        |
|          | 8.62±0.2ms                 | 8.63±0.06ms                                 | 1.00    | io.csv.ReadCSVParseSpecialDate.time_read_special_date('hm', 'python')                   |
|          | 7.29±0.2ms                 | 7.74±0.3ms                                  | 1.06    | io.csv.ReadCSVParseSpecialDate.time_read_special_date('mY', 'c')                        |
|          | 25.4±1ms                   | 25.0±0.1ms                                  | 0.99    | io.csv.ReadCSVParseSpecialDate.time_read_special_date('mY', 'python')                   |
|          | 3.54±0.2ms                 | 3.41±0.02ms                                 | 0.96    | io.csv.ReadCSVParseSpecialDate.time_read_special_date('mdY', 'c')                       |
|          | 9.54±0.06ms                | 9.49±0.1ms                                  | 1.00    | io.csv.ReadCSVParseSpecialDate.time_read_special_date('mdY', 'python')                  |
|          | 9.27±0.1ms                 | 10.1±0.3ms                                  | 1.09    | io.csv.ReadCSVSkipRows.time_skipprows(10000, 'c')                                       |
|          | 3.54±0.1ms                 | 3.48±0.09ms                                 | 0.98    | io.csv.ReadCSVSkipRows.time_skipprows(10000, 'pyarrow')                                 |
|          | 38.4±1ms                   | 37.3±1ms                                    | 0.97    | io.csv.ReadCSVSkipRows.time_skipprows(10000, 'python')                                  |
| +        | 14.5±0.1ms                 | 16.5±0.1ms                                  | 1.14    | io.csv.ReadCSVSkipRows.time_skipprows(None, 'c')                                        |
|          | 3.53±0.1ms                 | 3.57±0.1ms                                  | 1.01    | io.csv.ReadCSVSkipRows.time_skipprows(None, 'pyarrow')                                  |
|          | 57.9±0.7ms                 | 56.8±2ms                                    | 0.98    | io.csv.ReadCSVSkipRows.time_skipprows(None, 'python')                                   |
|          | 10.2±0.2ms                 | 10.6±0.3ms                                  | 1.04    | io.csv.ReadCSVThousands.time_thousands(',', ',', 'c')                                   |
|          | 128±5ms                    | 127±4ms                                     | 1.00    | io.csv.ReadCSVThousands.time_thousands(',', ',', 'python')                              |
|          | 9.74±0.2ms                 | 10.5±0.09ms                                 | 1.08    | io.csv.ReadCSVThousands.time_thousands(',', None, 'c')                                  |
|          | 56.2±0.9ms                 | 54.9±0.8ms                                  | 0.98    | io.csv.ReadCSVThousands.time_thousands(',', None, 'python')                             |
|          | 10.1±0.2ms                 | 10.9±0.2ms                                  | 1.08    | io.csv.ReadCSVThousands.time_thousands('|', ',', 'c')                                   |
|          | 122±3ms                    | 124±2ms                                     | 1.01    | io.csv.ReadCSVThousands.time_thousands('|', ',', 'python')                              |
|          | 10.1±0.2ms                 | 10.5±0.09ms                                 | 1.04    | io.csv.ReadCSVThousands.time_thousands('|', None, 'c')                                  |
|          | 52.9±2ms                   | 53.5±1ms                                    | 1.01    | io.csv.ReadCSVThousands.time_thousands('|', None, 'python')                             |
| +        | 1.43±0.06ms                | 4.04±0.1ms                                  | 2.83    | io.csv.ReadUint64Integers.time_read_uint64                                              |
| +        | 3.73±0.1ms                 | 6.36±0.05ms                                 | 1.71    | io.csv.ReadUint64Integers.time_read_uint64_na_values                                    |
| +        | 3.56±0.1ms                 | 6.05±0.1ms                                  | 1.70    | io.csv.ReadUint64Integers.time_read_uint64_neg_values                                   |
|          | 127±0.5ms                  | 126±0.5ms                                   | 0.99    | io.csv.ToCSV.time_frame('long')                                                         |
|          | 16.6±0.4ms                 | 16.5±0.2ms                                  | 1.00    | io.csv.ToCSV.time_frame('mixed')                                                        |
|          | 117±0.8ms                  | 116±0.8ms                                   | 0.99    | io.csv.ToCSV.time_frame('wide')                                                         |
|          | 9.36±0.2ms                 | 9.23±0.03ms                                 | 0.99    | io.csv.ToCSVDatetime.time_frame_date_formatting                                         |
|          | 3.84±0.1ms                 | 3.83±0.06ms                                 | 1.00    | io.csv.ToCSVDatetimeBig.time_frame(1000)                                                |
|          | 35.4±0.9ms                 | 35.2±0.8ms                                  | 0.99    | io.csv.ToCSVDatetimeBig.time_frame(10000)                                               |
|          | 348±9ms                    | 347±1ms                                     | 1.00    | io.csv.ToCSVDatetimeBig.time_frame(100000)                                              |
|          | 487±20ms                   | 487±4ms                                     | 1.00    | io.csv.ToCSVDatetimeIndex.time_frame_date_formatting_index                              |
|          | 157±4ms                    | 157±2ms                                     | 1.00    | io.csv.ToCSVDatetimeIndex.time_frame_date_no_format_index                               |
|          | 752±9ms                    | 737±10ms                                    | 0.98    | io.csv.ToCSVFloatFormatVariants.time_callable_format                                    |
|          | 832±10ms                   | 814±3ms                                     | 0.98    | io.csv.ToCSVFloatFormatVariants.time_new_style_brace_format                             |
|          | 918±10ms                   | 908±20ms                                    | 0.99    | io.csv.ToCSVFloatFormatVariants.time_new_style_thousands_format                         |
|          | 894±8ms                    | 894±10ms                                    | 1.00    | io.csv.ToCSVFloatFormatVariants.time_old_style_percent_format                           |
|          | 662±4ms                    | 657±3ms                                     | 0.99    | io.csv.ToCSVIndexes.time_head_of_multiindex                                             |
|          | 667±9ms                    | 661±10ms                                    | 0.99    | io.csv.ToCSVIndexes.time_multiindex                                                     |
|          | 659±10ms                   | 675±6ms                                     | 1.02    | io.csv.ToCSVIndexes.time_standard_index                                                 |
|          | 197±4ms                    | 190±1ms                                     | 0.97    | io.csv.ToCSVMultiIndexUnusedLevels.time_full_frame                                      |
|          | 16.9±0.1ms                 | 16.1±0.09ms                                 | 0.95    | io.csv.ToCSVMultiIndexUnusedLevels.time_single_index_frame                              |
|          | 18.8±0.5ms                 | 19.0±0.4ms                                  | 1.01    | io.csv.ToCSVMultiIndexUnusedLevels.time_sliced_frame                                    |
|          | 3.65±0.1ms                 | 3.80±0.06ms                                 | 1.04    | io.csv.ToCSVPeriod.time_frame_period_formatting(1000, 'D')                              |
|          | 3.76±0.04ms                | 3.82±0.03ms                                 | 1.02    | io.csv.ToCSVPeriod.time_frame_period_formatting(1000, 'h')                              |
|          | 35.8±0.3ms                 | 35.8±0.3ms                                  | 1.00    | io.csv.ToCSVPeriod.time_frame_period_formatting(10000, 'D')                             |
|          | 35.7±0.1ms                 | 35.9±0.2ms                                  | 1.01    | io.csv.ToCSVPeriod.time_frame_period_formatting(10000, 'h')                             |
|          | 1.13±0.02ms                | 1.17±0.02ms                                 | 1.04    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(1000, 'D')                      |
|          | 1.44±0.04ms                | 1.46±0.02ms                                 | 1.01    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(1000, 'h')                      |
|          | 10.2±0.2ms                 | 10.2±0.2ms                                  | 1.01    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(10000, 'D')                     |
|          | 12.9±0.3ms                 | 12.8±0.2ms                                  | 0.99    | io.csv.ToCSVPeriod.time_frame_period_formatting_default(10000, 'h')                     |
|          | 1.17±0.02ms                | 1.17±0.02ms                                 | 0.99    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(1000, 'D')             |
|          | 1.45±0.02ms                | 1.45±0.02ms                                 | 0.99    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(1000, 'h')             |
|          | 10.2±0.2ms                 | 10.3±0.2ms                                  | 1.00    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(10000, 'D')            |
|          | 13.1±0.4ms                 | 13.0±0.2ms                                  | 1.00    | io.csv.ToCSVPeriod.time_frame_period_formatting_default_explicit(10000, 'h')            |
|          | 5.40±0.02ms                | 5.41±0.02ms                                 | 1.00    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index(1000, 'D')                   |
|          | 5.49±0.05ms                | 5.49±0.02ms                                 | 1.00    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index(1000, 'h')                   |
|          | 50.5±0.3ms                 | 50.7±0.1ms                                  | 1.00    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index(10000, 'D')                  |
|          | 51.1±0.5ms                 | 51.0±0.3ms                                  | 1.00    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index(10000, 'h')                  |
|          | 1.16±0.02ms                | 1.16±0.02ms                                 | 1.00    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(1000, 'D')           |
|          | 1.44±0.03ms                | 1.43±0.02ms                                 | 1.00    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(1000, 'h')           |
|          | 9.64±0.2ms                 | 9.53±0.2ms                                  | 0.99    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(10000, 'D')          |
|          | 12.4±0.3ms                 | 12.2±0.3ms                                  | 0.99    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default(10000, 'h')          |
|          | 2.73±0.07ms                | 2.71±0.04ms                                 | 0.99    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(1000, 'D')  |
|          | 3.02±0.06ms                | 2.96±0.06ms                                 | 0.98    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(1000, 'h')  |
|          | 23.9±0.6ms                 | 24.1±0.4ms                                  | 1.01    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(10000, 'D') |
|          | 26.9±0.7ms                 | 27.1±0.5ms                                  | 1.01    | io.csv.ToCSVPeriodIndex.time_frame_period_formatting_index_default_explicit(10000, 'h') |

Verification performance overhead reduced from 2.83 to 2.47

Alvaro-Kothe · 2025-10-04T15:47:45Z

I managed to reduce the performance overhead of the verification a little bit. If the current solution is not satisfactory, I think it's possible to change the str_to_u?int64 function to manage float verification.

mroeschke · 2025-10-06T17:08:54Z

pandas/_libs/parsers.pyx

+        ignored_chars = b" +-"
+        digits = b"0123456789"
+        float_indicating_chars = b"eE"


Is it worth typing these in the cdef block?

I moved it and the test suite in io/parser/common passed. I don't recall how it was transpiled before. But I checked with the new changes and it's using native C types.

mroeschke · 2025-10-06T17:09:19Z

pandas/_libs/parsers.pyx

+
+            found_first_digit = False
+            j = 0
+            while word[j] != b"\0":


Same with this terminating char here?

I moved the null byte definition to cdef.

Alvaro-Kothe · 2025-10-06T18:44:30Z

pandas/_libs/parsers.pyx

+            const char *ignored_chars = " +-"
+            const char *digits = "0123456789"
+            const char *float_indicating_chars = "eE"
+            char null_byte = 0


Had to use = 0 instead of = '\0' because cython-lint doesn't let me use single quote, and it doesn't compile with double quote.

mroeschke · 2025-10-06T22:13:45Z

Thanks @Alvaro-Kothe

WillAyd · 2025-10-07T18:26:11Z

pandas/_libs/parsers.pyx

+                elif not found_first_digit and word[j] not in digits:
+                    # word isn't numeric
+                    return False
+                elif not found_first_digit and word[j] in digits:


<ctype.h> provides an isdigit function that would be preferable to use here, rather than rolling our own implementation

There's also strtof in <stdlib.h>` that you could use in lieu of most of this code

WillAyd · 2025-10-07T18:56:04Z

Any chance we've run the benchmarks for CSV reading? This might be expensive to take a second pass at columns like this

Alvaro-Kothe · 2025-10-07T19:39:51Z

Any chance we've run the benchmarks for CSV reading?

I ran them on my system and posted the results on #62542 (comment). The performance penalty was 2.5x slower from commit b3519c1.

@WillAyd I will open another PR to reduce the performance penalty caused by these changes.

I think it's possible to change the str_to_u?int64 function to manage float verification.

I think that the most performant approach is to verify if is float while it's parsing to integer in pandas/_libs/src/parser/tokenizer.c. What do you think?

WillAyd · 2025-10-07T19:59:08Z

Very nice - thanks for the thorough analysis @Alvaro-Kothe . I agree that its probably best to do this during tokenization, although I'm a bit wary of anything that is going to add performance overhead. Floats beyond 2^64 are going to suffer from precision loss, so it doesn't seem like general performance should take a step back for that

Alvaro-Kothe · 2025-10-07T20:29:03Z

so it doesn't seem like general performance should take a step back for that

Understood. This performance overhead for a big floating number edge case may not be worth it. I will prototype a tokenization solution to see if the performance overhead is negligible to what was before. If it isn't, I will prepare a PR to revert this one.

Alvaro-Kothe added 2 commits October 1, 2025 23:24

fix: make read_csv be able to read large numbers into float

1b6e3a2

docs(whatsnew): add entry in whatsnew

8908d84

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

Alvaro-Kothe added 7 commits October 2, 2025 10:39

test: test explicit float

2686655

hackish solution

4aa0ff1

test: add small float test

706503f

chore: return newline

544a257

docs: update whatsnew

62b49f9

early return

6c03279

test: add exponent tests

6aa8398

Alvaro-Kothe changed the title ~~BUG: make read_csv be able to read large numbers into float~~ BUG: make read_csv be able to read large floating numbers into float Oct 2, 2025

Alvaro-Kothe marked this pull request as ready for review October 2, 2025 16:10

Alvaro-Kothe added 4 commits October 2, 2025 16:44

refactor: simplify code

44b68bc

test: remove ids and add new test with no decimal separator

e16c71b

chore: add comments making explicit about precision loss

fc01097

Merge branch 'main' into fix/read_csv-large-num

93c94b5

mroeschke reviewed Oct 3, 2025

View reviewed changes

pandas/_libs/parsers.pyx Outdated Show resolved Hide resolved

mroeschke reviewed Oct 3, 2025

View reviewed changes

mroeschke added the IO CSV read_csv, to_csv label Oct 3, 2025

Alvaro-Kothe and others added 2 commits October 3, 2025 14:58

Update pandas/_libs/parsers.pyx

380e6ec

Co-authored-by: Matthew Roeschke <[email protected]>

fix: early return on non numeric chars

6b4cedb

perf: improve performance marginally

b3519c1

Verification performance overhead reduced from 2.83 to 2.47

Alvaro-Kothe added 2 commits October 4, 2025 12:54

perf: early return on decimal char

2e0af7a

Merge remote-tracking branch 'upstream/main' into fix/read_csv-large-num

0973086

mroeschke added this to the 3.0 milestone Oct 6, 2025

mroeschke reviewed Oct 6, 2025

View reviewed changes

Alvaro-Kothe added 3 commits October 6, 2025 15:22

fix: move variables declarations to cdef

c7ddf17

chore: use double quote in cython

a7ab941

chore: workaround double quote lint

8b57401

Alvaro-Kothe commented Oct 6, 2025

View reviewed changes

mroeschke approved these changes Oct 6, 2025

View reviewed changes

mroeschke merged commit db31f6a into pandas-dev:main Oct 6, 2025
41 checks passed

WillAyd reviewed Oct 7, 2025

View reviewed changes

Alvaro-Kothe deleted the fix/read_csv-large-num branch October 7, 2025 23:22

Uh oh!

BUG: make read_csv be able to read large floating numbers into float #62542

BUG: make read_csv be able to read large floating numbers into float #62542

Conversation

Alvaro-Kothe commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Implementation details

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Alvaro-Kothe commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alvaro-Kothe commented Oct 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Alvaro-Kothe Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mroeschke commented Oct 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd commented Oct 7, 2025

Uh oh!

Alvaro-Kothe commented Oct 7, 2025

Uh oh!

WillAyd commented Oct 7, 2025

Uh oh!

Alvaro-Kothe commented Oct 7, 2025

Uh oh!

Uh oh!

BUG: make `read_csv` be able to read large floating numbers into float #62542

BUG: make `read_csv` be able to read large floating numbers into float #62542

Alvaro-Kothe commented Oct 2, 2025 •

edited

Loading

Alvaro-Kothe commented Oct 3, 2025 •

edited

Loading

Alvaro-Kothe Oct 6, 2025 •

edited

Loading